Repository navigation
[minimax-m3]: integrate flydsl index_score kernel - #2269
ganyi1996ppo wants to merge 3 commits into
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
There was a problem hiding this comment.
🟡 Changes recommended
Two critical unresolved issues affect CP dispatch correctness and test collection.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Integrates AITER FlyDSL index-score decoding for MiniMax-M3 across native, vLLM, and SGLang attention paths.
Changes:
- Replaces Triton scoring with FlyDSL kernels.
- Adds persistent work-map metadata and context-parallel support.
- Expands score and parity test coverage.
File summaries
| File | Reviewed change |
|---|---|
tests/test_minimax_m3_index_topk.py |
Adds FlyDSL score-oracle and work-map tests. |
tests/model_ops/test_indexer_cp_parity.py |
Adds context-parallel scoring and parity coverage. |
atom/plugin/vllm/attention/minimax_m3_attnetion.py |
Wires score metadata into vLLM decoding. |
atom/plugin/vllm/attention/metadata.py |
Builds persistent vLLM work maps. |
atom/plugin/sglang/attention_backend/minimax_m3_sparse.py |
Hoists SGLang work-map creation. |
atom/model_ops/minimax_m3/sparse_attn.py |
Propagates sparse score metadata. |
atom/model_ops/minimax_m3/indexer_context_parallel.py |
Integrates FlyDSL scoring for context parallelism. |
atom/model_ops/minimax_m3/indexer_candidate_exchange.py |
Supports strided score layouts. |
atom/model_ops/minimax_m3/index_topk.py |
Adds FlyDSL scoring and dispatch-map APIs; critical optional-import issue remains (1 vote). |
atom/model_ops/attentions/aiter_attention.py |
Adds persistent score buffers; critical CP work-map sizing issue remains (2 votes). |
atom/model_ops/attention_mha.py |
Threads score metadata through native decoding. |
Review details
Suppressed comments (1)
atom/model_ops/attentions/aiter_attention.py:455
- The ubatch buffer is sized with the default
cp_world=1, cp_rank=0, butmake_sparse_decode_metadatalater fills it usingself._index_score_cp_world/self._index_score_cp_rank. Under indexer CP the flydsl map size/config is therefore not guaranteed to match the buffer, which can make the build overrun the ubatch buffer or reject the map. Use the same CP arguments here as in the persistent-buffer allocation above.
rows = index_score_work_map_size(
ub_max_bs,
self._index_score_max_block,
max_seqlen_qo,
self._num_idx_heads,
- Files reviewed: 11/11 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| rows = index_score_work_map_size( | ||
| self.max_bs, | ||
| self._index_score_max_block, | ||
| max_qlen, | ||
| num_head_k, |
| from aiter.ops.flydsl import ( | ||
| MiniMaxM3IndexScoreConfig, | ||
| minimax_m3_index_score_alloc, | ||
| minimax_m3_index_score_flydsl, | ||
| ) |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical work-map capacity issues and unresolved CP sizing and SGLang metadata findings block approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 4
Open (4)
Size work map for maximum capacity across batch sizes · New Size vLLM map for maximum captured-batch capacity · New [verified] This hard import makesindex_topkunimportable when the optional AITER runtime or the… When indexer CP is enabled,attention_mhasets the scorer'snum_idx_headsto the CP world size…
| rows = index_score_work_map_size( | ||
| self.max_bs, | ||
| self._index_score_max_block, | ||
| max_qlen, | ||
| num_head_k, | ||
| self._index_score_cp_world, | ||
| self._index_score_cp_rank, | ||
| ) |
| rows = max( | ||
| index_score_work_map_size( | ||
| max_bs, self._index_score_max_block, max_query_len, self._num_idx_heads | ||
| ) | ||
| for max_query_len in range(1, max(1, int(self.reorder_batch_threshold)) + 1) | ||
| ) |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical replay and empty-shard issues, dependency gating, and work-map sizing issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 6
Open (6)
Guard empty CP shards before page-table access · New Refill work map during CUDA-graph replay metadata updates · New Size vLLM map for maximum captured-batch capacity Size work map for maximum capacity across batch sizes [verified] This hard import makesindex_topkunimportable when the optional AITER runtime or the… When indexer CP is enabled,attention_mhasets the scorer'snum_idx_headsto the CP world size…
| return decode_index_score( | ||
| idx_q, index_cache, block_table, seq_lens, max_block or local, | ||
| max_query_len, heads, sm_scale, work_map, | ||
| cp_world=world_size, cp_rank=rank, | ||
| ) # fmt: skip |
| index_score_work_map=index_score_work_map_for_forward( | ||
| forward_batch, | ||
| "decode", | ||
| metadata.seq_lens, | ||
| max_block=metadata.block_table.shape[1], |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved work-map sizing, CP metadata, dependency, safety, and validation issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 7
Open (8)
Validate persistent output capacity before FlyDSL writes · New Refill work map during CUDA-graph replay metadata updates Guard empty CP shards before page-table access Size vLLM map for maximum captured-batch capacity Size work map for maximum capacity across batch sizes [verified] This hard import makesindex_topkunimportable when the optional AITER runtime or the… When indexer CP is enabled,attention_mhasets the scorer'snum_idx_headsto the CP world size… TBO work-map allocation assumes non-monotonic sizing · New
| from aiter.ops.flydsl import minimax_m3_index_score_work_map | ||
|
|
||
| return minimax_m3_index_score_work_map( | ||
| seq_lens, | ||
| max_block, | ||
| max_query_len, | ||
| num_idx_heads, | ||
| index_score_config(cp_world, cp_rank), | ||
| out=out, | ||
| ) |
| rows = index_score_work_map_size( | ||
| ub_max_bs, | ||
| self._index_score_max_block, | ||
| max_seqlen_qo, | ||
| self._num_idx_heads, |
|
try to slove the high comment from copilot? |
The flydsl index-score kernel landed beside the Triton decode scorers rather than in place of them: an env flag, a memoized support predicate, a `return None` decline ladder, and two kernels that had to be kept in agreement. This removes that half-state. `_decode_index_score_tiled_kernel` (TP) and `_context_score` (CP) are gone, `decode_index_score` calls flydsl unconditionally, and ATOM_USE_FLYDSL_INDEX_SCORE is deleted -- a kill switch with nothing to switch to is worse than no switch. Absence of the kernel is now an ImportError at module load naming the missing symbol, not a silent downgrade 57 sparse layers deep. Prefill is untouched. `_index_block_score_kernel` and its helpers stay: flydsl has no prefill scorer, and only the decode path was ever replaced. A missing work map is built on the spot rather than raising. That is correct but costs ~120 us of pure launch floor -- against a score kernel of 17-250 us -- so every production caller hoists it into per-step metadata instead. SGLang was the one caller with no such field and was therefore riding the Triton branch silently; it now hoists through `index_score_work_map_for_forward`, bounded by `block_table.shape[1]`. That bound is deliberate: the map's row count IS the kernel's grid, so it must be stable across a cudagraph replay, and this step's longest request is not. `build_minimax_m3_block_table` already slices to the static token-table width while capturing, so the bound is capture-stable for free. A map built against a different bound is caught by a row-count assert -- otherwise it silently walks rows that mean something else. Two consequences worth knowing before touching this code: - The score tensor cannot be read raw. The deleted Triton CP scorer overwrote every slot with -inf; flydsl writes only the blocks a request actually has, so a dead slot is uninitialised memory. Safe ONLY because `_pack_score_key` sends every masked lane to key 0, below any real candidate. `local_candidate_keys` now also takes the three score strides explicitly, because flydsl picks a feature-contiguous layout worth ~10% to it that a hardcoded `* LOCAL_BLOCKS` cannot address. - The tests lost their independent judge. With Triton gone, every kernel-vs-kernel comparison degenerated into kernel-vs-itself -- the CP parity suite compared against a `_tp_reference` that is now the same kernel. Both suites grew pure-torch oracles transcribed from the definition instead (base-2 scale, per-column causal cutoff `L - S + tok + 1`, max over the block's 128 tokens, K cast to Q's dtype BEFORE fp32 so fp8 rounding is reproduced). The CP oracle was checked to bite by mutating the remap `gp = p*world + rank`: 4 failures. Verified: 148 passed / 1 skipped across index_topk, envs and CP parity (the skip is pre-existing -- aiter's small-k selector is not installed here). Server smoke on the CP arm, one 5,610-token completion, traces 912 launches of m3_index_score_S1_H4_fp8_L1_cp4r0 on rank 0 and _cp4r1 on rank 1 (57 sparse layers x 16 decode tokens, the rank folded in as a compile-time constant), and zero of either deleted kernel. Co-Authored-By: Claude <noreply@anthropic.com>
…strides
Two defects in the flydsl index-score integration, both found by
measurement after rebasing onto main.
1. The ubatch work-map buffer was sized with the default cp_world=1
while `_build_ubatch_metadata` builds it with the real CP pair. The
shard's dispatch is denser than the whole context's, so the CP map is
LONGER, not shorter -- 128 rows against 64 at bs=8, max_block=64 --
and the build rejects the buffer:
ValueError: work_map: expected packed [at least 128, 2] int32
So ubatch plus indexer CP could not start. It fails loudly rather
than writing out of bounds, which is why it never showed as
corruption. The non-ubatch buffer next to it already passed the pair;
this just does the same.
2. The three score strides were `tl.constexpr`. `S_HEAD` is
`tokens * local`, so it moves with the batch, and specializing on it
cost a ~430ms Triton recompile for every distinct decode batch size:
8 of 8 sizes recompiled with a cold Triton cache, 0 of 8 after. That
is the same reason `LOCAL_BLOCKS` and `GLOBAL_BLOCKS` are already
runtime arguments in this kernel. Triton still folds the
divisible-by-16 and equal-to-1 cases, which is where the addressing
win is; nothing about the layout handling changes.
tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py
and tests/test_paged_attention_dispatch.py pass (216 passed, 21 skipped).
Not verified end to end on a server: main now imports `aiter.topk_select`
(#2244, #2314) and the aiter installed here does not have it, so the
engine cannot start from this base at all -- with or without this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cef632d to
757ed8c
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
One or more issues must be addressed before approval.
Review effort: Lite
Findings: 9
Open (10)
Pin required AITER companion revision in build workflows · New Handle or reject fnuz FP8 caches on gfx942 · New Validate persistent output capacity before FlyDSL writes Refill work map during CUDA-graph replay metadata updates Guard empty CP shards before page-table access Size vLLM map for maximum captured-batch capacity Size work map for maximum capacity across batch sizes [verified] This hard import makesindex_topkunimportable when the optional AITER runtime or the… When indexer CP is enabled,attention_mhasets the scorer'snum_idx_headsto the CP world size… TBO work-map allocation assumes non-monotonic sizing
| from aiter.ops.flydsl import ( | ||
| MiniMaxM3IndexScoreConfig, | ||
| minimax_m3_index_score_alloc, | ||
| minimax_m3_index_score_flydsl, | ||
| ) |
| minimax_m3_index_score_flydsl( | ||
| idx_q, | ||
| index_kv_cache, | ||
| block_table, | ||
| seq_lens, | ||
| max_query_len, | ||
| num_idx_heads, | ||
| sm_scale, | ||
| max_block, | ||
| out=score, | ||
| cfg=cfg, | ||
| work_map=work_map, |
|
Superseded by #2493, which is the same work on a later branch ( #2493 covers every file here except Closing here so there is one PR on this topic. Review continues on #2493. |


[MiniMax-M3] Score the lightning indexer with aiter's FlyDSL kernels
Branch:
ganyi/m3-flydsl-index-score-integration— one commit on top ofmain(a526f0d55), 9 files, +1161/−395.What this changes
MiniMax-M3's lightning indexer scores every 128-token block of index keys and keeps the top-k. Both phases did that in Triton. This routes them to aiter's FlyDSL MFMA kernels.
No accuracy decision is being made here. The scorers compute the same quantity — the max of
q·kover each 128-token block — and aiter'salloc_scoredeliberately writes the layout the Triton kernel wrote, so everything downstream is blind to the choice. They are not bit-exact: the reductions run in a different order, so a tie sitting exactly on the top-k cut can resolve to a different block. gsm8k shows no detectable effect from that (8 reps per arm, below)._decode_index_score_tiled_kernel(Triton)minimax_m3_index_score(FlyDSL) — the only scorer_context_score(Triton)cp_world/cp_rank_index_block_score_kernel(Triton)ATOM_M3_PREFILL_INDEX_SCOREDecode keeps no fallback. The two Triton decode kernels are deleted and the aiter import is hard, so an aiter that lacks the kernel is an
ImportErrorat module load rather than a silent downgrade discovered 57 layers into a decode step.Performance
End-to-end: the controlled overnight sweep
This is the measurement the PR rests on. MiniMax-M3-MXFP4, TP4 on one GPU quad shared by every arm, indexer CP on,
--kv_cache_dtype fp8,--block-size 128, eagle3 with--num-speculative-tokens 7and forced acceptance length 5, aiperf agentic traces (inferencex-agentx-mvp) at concurrency 25, 900 s profiling window per run.Each run is standalone (server up → client → server down) and the arm order rotates by cycle, so a run's position in the night is not confounded with its arm. 7 complete runs, all stall-free (zero seconds of engine silence ≥20 s inside the window).
The two candidate arms map exactly onto this PR's two configurations:
flydsl-noshuffle≡ what this PR ships by default on an fp8 cache (both scorers FlyDSL)flydsl-decode≡ATOM_M3_PREFILL_INDEX_SCORE=triton(decode scorer only)triton(baseline)flydsl-decodeflydsl-noshuffleWhole-run throughput carries ~4 points of request-mix noise — two runs of the same arm once shared 99.8% of their turns with bit-identical total OSL yet differed 5.2%, because the 29 unshared turns averaged 2,547 tokens each. So the headline number is the mix-free one: pair every request on
(conversation_id, turn_index), which is the same trace turn with the same input and the same trace-fixed output length served twice, and take the geometric mean of the per-request ratios.flydsl-decodeflydsl-noshuffleRead this as ~4–5% on interactivity and request latency, with the decode scorer carrying roughly three quarters of it and the prefill scorer adding ~1 point.
What the data does not support:
flydsl-noshuffleand +0.59% forflydsl-decode; the paired column reverses that ordering (−2.29% vs −3.84%). Within-arm TTFT spread is ±16%, and bootstrap CIs on agentic TTFT straddle zero at every percentile. Treat TTFT as unchanged.autoenables the prefill scorer — but this is a lean, not a separation.Prefill kernel microbenchmark — why
autoreads the cache dtypeFrom the earlier direct kernel measurement, not re-run for this PR. One index head (prefill stays TP even under indexer CP), 32768-token chunk, plain (unshuffled) cache, bf16 query throughout — only the cache dtype moves:
The cliff is
k_lds, the path that stages K through LDS and so reads a quarter of the K bytes. It is gated onfp8 and fp8_mfma and k128, andfp8is read off the cache, not the query:Without
k_ldsthe kernel takes a register path that loses to Triton outright. SoATOM_M3_PREFILL_INDEX_SCORE=auto(the default) picks FlyDSL on an fp8 index cache and Triton on bf16 — and falls back to Triton on an aiter that predates the kernel, since it is newer than the aiter ATOM pins and an aiter without it is still a working M3.On M3 the index cache dtype follows
--kv_cache_dtype(config.py:2589), so in practice--kv_cache_dtype fp8is what turns this on.The two explicit values (
flydsl,triton) never fall back; asking forflydslon an aiter that lacks it raises. An arm that asked for FlyDSL and silently ran Triton is a wrong measurement, not a degraded one — that has already cost one wrong conclusion on this model. Use a name, notauto, when benchmarking.Earlier measurements, for context
From memory of prior runs on a different configuration; listed because they are what motivated the work, not as evidence for this PR.
Accuracy — gsm8k, 8 reps per arm
Run on a separate server with no speculation at all, not against the throughput runs above: those pin the acceptance length, which makes the generated text meaningless and any score computed from it worthless.
Both arms: MiniMax-M3-MXFP4, TP4 on the same GPU quad, indexer CP on (the
out=contiguity fix lives in the CP decode path, so an accuracy run with CP off would never execute the line most worth checking),--kv_cache_dtype fp8,--hf-overrides '{"use_index_cache": true, "index_topk_freq": 4}'. The baseline is anorigin/mainworktree — not a flag on this branch, because decode has no Triton scorer left to select.lm_eval --tasks gsm8k --num_fewshot 5vialocal-chat-completions,max_tokens=8192,temperature=0.origin/main(Triton)Δ = −0.30 points, Welch t = 0.98, df = 13.9 (p ≈ 0.34) — not distinguishable from noise. Each arm's own spread across reps (1.67 and 1.74 points) is more than five times the difference between them.
This needed all 8 reps to see. At 4 reps per arm the same experiment read −1.04 points with 3 of 4 baseline reps above every candidate rep, which looks like a regression; the next 4 candidate reps averaged 94.64% against the first 4 at 93.55%, i.e. one arm moved 1.09 points against itself. Anyone re-running this should budget 8 reps a side before reading a sign off it.
Do not compare these against the older
gsm8k_v3numbers (93.9 – 94.8%). Those were run withoutuse_index_cache/index_topk_freq=4, andindex_topk_freq=4is itself an approximation — the configurations are not comparable.The work map, and where it is built
The FlyDSL decode kernel takes a packed dispatch map so a ragged batch does not pay for its holes. Building it is ~15 trivial device ops whose cost is almost entirely launch floor: ~120 µs a call, against a score kernel of 17–250 µs. Per sparse layer that costs more than the kernel it feeds.
It is therefore built once per decode step in the metadata builder (
MiniMaxM3SparseMetadata.index_score_work_map) and threaded to both the plain and the indexer-CP decode paths. Two properties this forces:n_valid_column_per_rowalready is.ceil(max_model_len / 128)— never the batch's longest request, which would reshape the map under a fixed grid. Under indexer CP it is this rank's local bound,ceil(that / world), because the shard is what the kernel is dispatched against.Omitting the map is still correct —
decode_index_scorebuilds its own — which is what the test and plugin paths do.Two aiter contracts, now checked instead of assumed
alloc_scorereturns a permuted view. Pasttotal_q * num_idx_heads >= 16aiter allocates[max_block, total_q, H]and hands back a feature-contiguous[H, total_q, max_block]permutation, which coalesces the scorer's store. That is every qlen>1 decode at CP world 4.local_candidate_keysrejects a non-contiguous score outright — exactly the check aiter's own docstring asks callers to make. The CP path now requests the layout it needs through a newout=ondecode_index_score.Both layouts carry bit-identical values (verified directly: max abs diff 0.0), so this costs the store's coalescing and nothing else, and it beats copying after the fact.
_select_with_aiterdeclines on the same condition, since its flattening is a.view.Block tables must cover
max_block. A recent aiter rejects a table narrower than the bound rather than reading past the row. Production never produces that pair — the engine sizes block tables frommax_model_len, the same quantity the bound comes from — so the tests were widened to match production rather than the check relaxed. Under CP the widening is byworld * pad, since the bound is local and the table is global.Tests
The wider tracked M3 + attention suite (
test_minimax_m3_*,test_sparse_attn_prefill_composition,test_paged_attention_dispatch,test_attn_family,test_mla_index_cache,test_mtp_index_share,test_sparse_indexer_chunk) was run against anorigin/mainworktree control on the same set: both green, 478 vs 474 passed, differing only by the shuffled-layout tests this PR removes.Not in this PR
The shuffled index-cache layout (
ATOM_M3_INDEX_SHUFFLED) and the fp8 index query (ATOM_M3_INDEX_FP8_Q), both deleted along with their env vars.The shuffled writer measured as a regression in its own right (−12.9% throughput, +17.4% ITL), and in the overnight sweep the shuffled arm came in below the plain one. With aiter no longer gating
k_ldson the layout, there is nothing left for it to buy.The fp8 query goes with it because it changes decode scores: decode's default bf16 MFMA lifts K to Q's precision rather than rounding Q down, so an fp8 Q there is not free, and a tie reordered near the top-k cut changes which blocks get selected. That is an accuracy decision and does not belong in a performance PR. (In prefill an fp8 Q is bit-identical, since
fp8_mfmawas rounding Q at staging anyway — which is why nothing is lost on that side.) Say the word if you want it kept as an opt-in.GLM-5.2 DCP sparse prefill — a separate PR, as discussed.
🤖 Generated with Claude Code